Imaging apparatus and distance measurement method

ABSTRACT

An imaging apparatus includes: an imaging unit; a focal range control unit configured to cause the imaging unit to capture a first image and a second image which have mutually different focal ranges, changing a focal position of the imaging unit; a reference image generation unit configured to generate, using the first image and the second image, a reference image to be used as a blur standard; and a distance measurement unit configured to measure a distance to the subject based on a difference in blur degrees between the reference image and each of the first image and the second image. The focal ranges of the first image and the second image are independent of each other, and an out-of-focus space is provided between the focal ranges.

TECHNICAL FIELD

The present invention relates to imaging apparatuses and distancemeasurement methods, particularly to an imaging apparatus in use formeasuring a distance to a subject using a plurality of captured images,and a distance measurement method performed using the apparatus.

BACKGROUND ART

Various techniques have been suggested for measuring, without contact, adepth of a three-dimensional scene, that is, a distance to each subject.Such techniques can be largely classified into an active technique andpassive technique. In the active technique, the subject is irradiatedwith infrared rays, ultrasound, or laser, so as to calculate a distancebased on a length of time until a reflected wave returns or an angle ofthe reflected wave. In the passive technique, the distance is calculatedbased on an image of the subject. Particularly, in the case of using acamera to measure the distance to the subject, the passive techniquewhich does not require an apparatus for emitting infrared rays and so onis widely used.

Various passive techniques have been suggested, one of which is referredto as Depth from defocus (hereinafter, referring to as “DFD”) which is atechnique to measure the distance based on a blur generated by focus(focal position) change. The DFD has features such as not requiring aplurality of cameras, allowing distance measurement using a small numberof images, and so on.

The DFD is a distance measurement technique using a blur of an image.However, there is a problem that it is extremely difficult to determine,from the captured image alone, whether the blur in the captured imagewas caused by change in lens focus or whether an original image whichrepresents a state without lens-derived blur has a blurred texture fromthe beginning.

To deal with this, Patent Literature 1 discloses a distance measurementmethod independent from a spectral component of the original image, inwhich a ratio of a spatial frequency spectrum in each of a plurality ofcaptured images is compared with a ratio of the spatial frequencyspectrum of the blur corresponding to the depth of the scene.

On the other hand, Patent Literature 2 discloses a technique to obtain areference image corresponding to the original image by capturing a largenumber of images by changing focus and extracting only focused portionsof the images. The distance is measured by configuring a scale space inwhich various blurs are convolved into this reference image, andcomparing the reference image and captured image on the scale space.

CITATION LIST Patent Literature

-   [Patent Literature 1]-   Japanese Patent Application Publication No. 11-337313-   [Patent Literature 2]-   US Patent Application Publication No. 2007/0019883, Specification

Non Patent Literature

-   [Non Patent Literature 1]-   H. Nagahara, S. Kuthirummal, C. Zhou, S. K. Nayer, “Flexible Depth    of Field Photography”, European Conference on Computer Vision    (ECCV), October, 2008

SUMMARY OF INVENTION Technical Problem

However, to facilitate measurement, the technique in Patent Literature 1is configured with a mask having a special structure inserted into anaperture, so that a zero point appears periodically in the spatialfrequency spectrum of the blur. However, this has a problem of causing adecrease in amount of incident light. It is therefore, necessary toeither increase sensitivity of the imaging device or increase anexposure-time period. However, the former causes an increase in noise ofthe captured image, and the latter causes a blur in the subject. Thesedisturb spectral components of the subject to decrease accuracy indistance measurement.

On the other hand, in the technique disclosed in Patent Literature 2, awidth of a depth of field is by far narrower than a distance measurementrange unless the aperture is significantly stopped down, thus requiringa large number of images to be captured so as to obtain the referencepicture. This offsets an advantage of the DFD which allows distancemeasurement using a small number of images. In addition, suchsignificant stopping down of the aperture allows obtaining the referenceimage from a small number of images, but also decreases the amount ofincident light, so that the technique in Patent Literature 2 has thesame problem as the problem of the technique disclosed in PatentLiterature 1.

An object of the present invention which is conceived in view of theproblem above is to provide an imaging apparatus and distancemeasurement method which allow stable distance measurement from a smallnumber of captured images.

Solution to Problem

In order to achieve the aforementioned object, an imaging apparatusaccording to an embodiment of the present invention includes: an imagingunit configured to image a subject to generate an image; a focal rangecontrol unit configured to cause the imaging unit to capture n imageshaving mutually different focal ranges, changing a focal position of theimaging unit, where n is an integer greater than or equal to 2; areference image generation unit configured to generate, using the nimages, a reference image to be used as a blur standard; and a distancemeasurement unit configured to measure a distance to the subject basedon a difference in blur degrees between the reference image and each ofthe n images, in which the focal ranges of the n images are independentof each other, and an out-of-focus space is provided between the focalranges.

With this configuration, in the imaging apparatus according to anembodiment of the present invention, the image is captured with a focalposition being changed, thereby obtaining an image which has the focalrange broader than average images without stopping down an aperture.This enables the imaging apparatus according to the embodiment of thepresent invention to obtain the reference image from a small number ofimages. In addition, the focal ranges of the respective images areindependent of each other. Accordingly, an image which has anapproximately uniform blur can be generated with respect to the distanceto the subject, using a plurality of images. Therefore, in the imagingapparatus according to the embodiment of the present invention, thereference image which has high accuracy can be obtained with a simplemethod. As mentioned above, in the imaging apparatus according to theembodiment of the present invention, a stable distance measurement canbe achieved from a small number of captured images.

The imaging unit may include: an imaging device; and a lens whichcollects light into the imaging device, and the focal range control unitmay be configured to change the focal position by changing a distancebetween the imaging device and the lens at a constant speed.

With this configuration, a depth of field is extended by a changeddistance between the imaging device and the lens, thereby enabling thedistance measurement in a broad range with a small number of images.

The n images may have a same exposure-time period.

With this configuration, noise included in the n images which havemutually different focal ranges can be made equally, thereby improvingaccuracy in distance calculation.

The focal range control unit may be configured to change the distancebetween the imaging device and the lens at the constant speed during atime period from initiation to completion of the imaging of the n imagesby the imaging unit.

With this configuration, control in changing the focal position can beeasily performed.

The imaging unit may include: a lens; n imaging devices arranged to havemutually different optical path lengths from the lens; and a beamsplitter which splits a light beam from the lens into light beams forthe respective n imaging devices. The focal range control unit may beconfigured to cause the n imaging devices to simultaneously capture therespective n images, changing the focal positions of the n imagingdevices simultaneously, during a same time period.

With this configuration, the n images having mutually different focalranges can be simultaneously captured, thereby reducing a time periodrequired for entire processing. Furthermore, each of the images has adepth of field which is continuously extended. Therefore, the distancemeasurement can be performed in a broad range with a small number ofimages.

The imaging unit may include: a lens; n imaging devices; and a selectionunit configured to allow a light beam from the lens to selectively enterone of the n imaging devices. The focal range control unit may beconfigured to sequentially select the n imaging devices, and to causethe selection unit to allow the light beam to selectively enter theselected imaging device, thereby causing each of the n imaging devicesto capture a corresponding one of the n images.

With this configuration, the n images having mutually different focalranges can be simultaneously captured, thereby reducing the time periodrequired for the entire processing. Furthermore, the images have thefocal ranges which are discontinuous to one another, causing each of theimages to have a blur shape based on which the distance measurement iseasily performed by DFD algorithm.

The reference image generation unit may be configured to generate anaverage image of the n images, and to generate the reference image usingthe average image.

With this configuration, the focal ranges of the n images areindependent of each other. Accordingly, the average image of the imageshas the uniform blur with respect to the distance to the subject.Therefore, the reference image having high accuracy can be obtained withthe simple method.

The reference image generation unit may be configured to generate thereference image by performing, on the average image, a deconvolutionoperation by a single point spread function.

It should be noted that the present invention can be achieved not onlyas a distance measurement apparatus but also as a distance measurementmethod including, as steps, processing implemented by thedistinguishable units included in the distance measurement apparatus.The present invention can also be achieved as a method for controllingthe imaging apparatus, or as a program which causes a computer toexecute such distinguishable steps. In addition, it is needless to saythat such a program can be distributed via a non-transitory computerreadable recording medium including a Compact Disc-Read Only Memory(CD-ROM) and so on or a transmission network such as the Internet.

Furthermore, the present invention can be also achieved as asemiconductor integrated circuit (LSI) which realizes a part of or allof function of the imaging apparatus.

Advantageous Effects of Invention

As described above, the present invention can provide an imagingapparatus and a distance measurement method which can achieve a stabledistance measurement from a small number of captured images.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram which shows a configuration of an imaging apparatusaccording to an embodiment of the present invention.

FIG. 2 is a diagram which schematically shows a state of lightcollection when a focal position is changed, according to the embodimentof the present invention.

FIG. 3 is a flowchart which shows a flow of operation of the imagingapparatus according to the embodiment of the present invention.

FIG. 4 is a graph which shows a changing state of the focal position,according to the embodiment of the present invention.

FIG. 5 is a graph which shows blur amounts in a first image and a secondimage, according to the embodiment of the present invention.

FIG. 6A is a diagram which shows a texture of a scene according to theembodiment of the present invention.

FIG. 6B is a diagram which shows a distance to a subject in theaccording to the embodiment of the present invention.

FIG. 7A is a diagram which shows a result of distance measurementperformed using two images having a continuous focal range, according tothe embodiment of the present invention.

FIG. 7B is a diagram which shows a result of distance measurementperformed using two images having discontinuous focal ranges, accordingto the embodiment of the present invention.

FIG. 8A is a table which shows a relationship between a space of thefocal ranges and RMS of the result of the distance measurement,according to the embodiment of the present invention.

FIG. 8B is a graph which shows a relationship between the space of thefocal ranges and RMS of the result of the distance measurement,according to the embodiment of the present invention.

FIG. 9A is a diagram which shows a configuration of an imaging unitaccording to Variation 1 of the present invention.

FIG. 9B is a graph which shows a changing state of a focal positionaccording to Variation 1 of the present invention.

FIG. 10 is a diagram which shows a configuration of an imaging unitaccording to Variation 2 of the present invention.

FIG. 11A is a diagram which shows an operation of the imaging unitaccording to variation 2 of the present invention.

FIG. 11B is a diagram which shows the operation of the imaging unitaccording to Variation 2 of the present invention.

FIG. 11C is a diagram which shows the operation of the imaging unitaccording to Variation 2 of the present invention.

FIG. 12 is a graph which shows a changing state of a focal positionaccording to Variation 2 of the present invention.

DESCRIPTION OF EMBODIMENTS

Hereinafter, embodiments of the present invention is described referringto drawings. It should be noted that the embodiments described beloweach represent preferred embodiments of the present invention. A numeralvalue, a shape, a material, a component, an arrangement position andconnection condition of the components, a step, and a flow of steps, andthe like described in the embodiments are merely examples, and are notintended to limit the present invention. The present invention islimited only by the scope of the claims. Accordingly, the componentwhich is not defined in an independent claim representing the broadestconcept of the present invention is described as not being necessarilyrequired for achieving the object of the present invention butconstituting a more preferred embodiment.

In an imaging apparatus according to the embodiments of the presentinvention, a plurality of images are captured each of which has anindependent focal range. In addition, a space is provided between eachof the plural focal ranges of the plural images. This allows the imagingapparatus to generate an image which has an approximately uniform blurwith respect to a subject distance, using these plural images. Asmentioned above, in the imaging apparatus according to the embodimentsof the present invention, stable distance measurement can be achievedfrom a small number of captured images.

FIG. 1 is a diagram which shows a configuration of an imaging apparatus10 according to the embodiment of the present invention. The imagingapparatus 10 shown in FIG. 1 captures an image and measures a distancebetween the imaging apparatus 10 and a subject captured in the image.The imaging apparatus 10 includes an imaging unit 11, a focal rangecontrol unit 12, a reference image generation unit 13, and a distancemeasurement unit 14.

The imaging unit 11 includes a lens unit in which a lens 21 collectinglight is incorporated, and an imaging device 22 such as a charge coupleddevice (CCD) sensor or a complementary metal oxide semiconductor (CMOS)sensor. The imaging unit 11 has a function to generate an image bycapturing an image of a subject.

The focal range control unit 12 has a function to control the lens unitof the imaging unit 11, so that a focal position and a depth of fieldare controlled. Specifically, the focal range control unit 12 causes afocal position to be changed by changing a distance between the lens 21and the imaging device 22. More specifically, the focal range controlunit 12 changes the distance between the lens 21 and the imaging device22 by moving one of the lens 21 and the imaging device 22 or both lens21 and the imaging device 22. For example, the focal range control unit12 causes autofocus mechanism incorporated in the lens unit to operateat a specific pattern or switches particular optical elements with oneanother. It should be noted that the lens unit may include a pluralityof lenses. In this case, the focal range control unit 12 may move one ormore lenses among the plural lenses. The distance between the lens andthe imaging device corresponds to a distance from a principle point ofthe lens to be moved or the plural lenses, to the imaging device, forexample.

Furthermore, the focal range control unit 12 causes the imaging unit 11to capture a plurality of images (a first image 31 a and a second image31 b) having mutually different focal ranges and depths of field,causing a focal position of the imaging unit 11 to be changed. Here,each of the focal ranges of the plural images is independent of eachother, and an out-of-focus space is defined between each of the focalranges.

Although an example in which two images are used that have differentfocal ranges and depths of field is described in the following, three ormore images may be used in the example.

The reference image generation unit 13 generates a reference image 32 tobe used as a blur standard, using the first image 31 a and the secondimage 31 b which are generated by operation of the focal range controlunit 12 to have mutually different focal positions and depths of field.Specifically, the reference image 32 is an image which estimates a statewithout blurs by the lens 21.

The distance measurement unit 14 performs distance measurement based onthe DFD technique, using the first image 31 a, the second image 31 b,and the reference image 32. Specifically, the distance measurement unit14 measures a distance to the subject from a difference in blur degreesof images between the reference image 32 and each of the first image 31a and the second image 31 b.

Next, technique of extending the depth of field of the captured image isdescribed. Generally, a width of depth of field is defined as follows.First, hyperfocal distance is described. The hyperfocal distance is adistance at which, when the lens is focused at the distance, an areabeyond the distance (on a farther side from the lens) up to an infinitedistance is judged to be in focus. The hyperfocal distance h can beapproximated using Expression 1 below when f is a focal distance of thelens, F is an F-number of the lens, and c is a size of a permissiblecircle of confusion which indicates a smallest detectable blur.

$\begin{matrix}\lbrack {{Math}.\mspace{14mu} 1} \rbrack & \; \\{h = \frac{f^{2}}{Fc}} & {{Expression}\mspace{14mu} 1}\end{matrix}$

Expression 2 and Expression 3 below represent the depth of field at thetime of focusing at the distance s, when DN is the depth of field in afront side of the distance s and Df is the depth of field in a rear sideof the distance s.

$\begin{matrix}\lbrack {{Math}.\mspace{14mu} 2} \rbrack & \; \\{D_{n} = \frac{( {s - f} )^{2}}{h + s - {2\; f}}} & {{Expression}\mspace{14mu} 2} \\{D_{f} = \frac{( {s - f} )^{2}}{h - s}} & {{Expression}\mspace{14mu} 3}\end{matrix}$

According to the expression above, in the case of fixed focal distance,the width of the depth of field can be changed only by stopping down anaperture.

In contrast, various techniques have been suggested to extend the widthof the depth of field without stopping down the aperture. These arereferred to as extended depth of field (hereinafter, “EDOF”). SpecificEDOF techniques are described below.

A simplest EDOF technique is to capture a plurality of images byshifting the focal position little by little, and extracting thesynthesizing focused portions from these images. The technique is alsoused in Patent Literature 2.

Whereas, Non-Patent Literature 1 discloses a technique of changing focusduring exposure to realize the same effect as the effect produced bysynthesizing a large number of images.

FIG. 2 is a diagram which schematically shows a state of lightcollection when the focal position is changed. As shown in FIG. 2, it isassumed that b1 is a position of the imaging device 22 (an image planeposition) focused at a subject position a1, and b2 is a position ofimaging device 22 focused at a subject position a2. According to thelens formula, the position of the imaging device 22 focused at anarbitrary subject position between a1 and a2 is always located betweenb1 and b2. Accordingly, when the imaging device 22 moves from b1 to b2during exposure, with respect to a1, the blur gradually becomes larger,staring from a point at which the imaging device 22 is focused at a1,and such blurs are integrated to appear overlapped in the image. On theother hand, with respect to a2, the imaging device 22 is graduallybrought into focus starting from a point at which the blur is large, andsuch blurs still appear overlapped in the image.

This is described in further detail, using an equation. It is assumedthat Expression 4 and Expression 5 below express a shape of a pointspread function (hereinafter described as “PSF”) which represents astate of blur in the optical system.

$\begin{matrix}\lbrack {{Math}.\mspace{14mu} 3} \rbrack & \; \\{{{PSF}( {r,u,{\Delta\; v}} )} = {\frac{2}{{\pi({gb})}^{2}}{\exp( {- \frac{2\; r^{2}}{({gb})^{2}}} )}}} & {{Expression}\mspace{14mu} 4} \\{b = {\frac{a}{v}{{\Delta\; v}}}} & {{Expression}\mspace{14mu} 5}\end{matrix}$

However, f represents a focal distance of the optical system, arepresents the diameter of an aperture of the optical system, urepresents a subject distance, v represents an image plane positiondefined by a lens formula 1/f=1/u+1/v, Δv is an amount of movement ofthe image plane from v, R represents a distance from a blur center, andg represents a constant. Here, the image plane position is a position ofthe imaging device based on the lens. In other words, the image planeposition corresponds to a distance between the lens and the imagingdevice.

Since each of Expression 4 and Expression 5 is a function of the amountof movement Δv, in the case of Δv changing from time 0 to time T inaccordance with the function V(t), an ultimate PSF can be defined asExpression 6 below.

$\begin{matrix}\lbrack {{Math}.\mspace{14mu} 4} \rbrack & \; \\{{{IPSF}( {r,u} )} = {\int_{0}^{T}{{{PSF}( {r,u,{V(t)}} )}\ {\mathbb{d}t}}}} & {{Expression}\mspace{14mu} 6}\end{matrix}$

Here, assuming that V(t) is uniform motion, that is, represented byV(t)=v0+st, Expression 6 can be solved as Expression 7 below.

$\begin{matrix}{\mspace{79mu}\lbrack {{Math}.\mspace{14mu} 5} \rbrack} & \; \\{{{IPSF}( {r,u} )} = {\frac{uf}{( {u - f} )\sqrt{2\;\pi}{rasT}}( {{{erfc}( \frac{r}{\sqrt{2}{{gV}(0)}} )} + {{erfc}( \frac{r}{\sqrt{2}{{gV}(T)}} )}} )}} & {{Expression}\mspace{14mu} 7}\end{matrix}$

Here, erfc(x) is a complementary error function, and b(t) is a diameterof the blur at the time t. Non-Patent Literature 1 discloses that thePSF that can be obtained from Expression 7 has an approximately constantblurred shape regardless of the distance between the distance V(0) toV(T). In other words, a starting position v+V(0) and an ending pointv+V(T) of the image plane position are changed, so that a range of thesubject distance can be changed in which the blur is constant.

Next, a DFD method using a reference image is described. Expression 8below shows a relationship between a reference image R and a capturedimage I.[Math. 6]I(x,y)=R(x,y)*h(x,y,d(x,y))  Expression 8

Here, h represents PSF at a position (x, y), and d(x, y) represents thesubject distance at the position (x, y). In addition, * in theexpression represents a convolution operation. For example, as shown inExpression 4 and Expression 5, PSF varies according to the subjectdistance. Accordingly, when a plurality of subjects are present atdifferent distances, images can be obtained into each of which aconvolution operation is performed with PSF that varies according to theposition of each image, as the captured images.

Here, it is assumed that the PSF that corresponds to each of the subjectdistances d1, d2, . . . , dn is h(x, y, d1), h(x, y, d2), . . . , andh(x, y, dn). For example, when the subject in the reference image R(x,y) is located at the distance d1, the captured image I(x, y) is equal toan image in which the convolution operation is performed with H(x, y,d1) into R(x, y), and a difference occurs between the captured imageI(x, y) and an image in which the convolution operation is performedwith the PSF corresponding to another subject distance into R(x, y).Therefore, the subject distance d(x, y) can be calculated bysequentially comparing the difference between each of the plural imagesin which the convolution operation is performed with each PSF into R(x,y) and the captured image, and finding the distance corresponding to aPSF having the smallest distance. Specifically, the subject distanced(x, y) can be calculated according to Expression 9 below.

$\begin{matrix}\lbrack {{Math}.\mspace{14mu} 7} \rbrack & \; \\{{{d( {x,y} )} = {\underset{d}{\arg\;\min}( {{I( {x,y} )} - {{R( {x,y} )}*{h( {x,y,d} )}}} )^{2}}}( {{d = d_{1}},d_{2},\ldots\mspace{14mu},d_{n}} )} & {{Expresion}\mspace{14mu} 9}\end{matrix}$

In practice, in order to reduce the influence of noise included in thecaptured image I, the image is segmented into blocks to calculate atotal sum of errors within the block, followed by determining, as thedistance for the all blocks, the distance at which the number of errorsis smallest. This allows the distance measurement to be performed morereliably.

Next, with reference to FIGS. 3, 4, and 5, described is a flow ofprocessing in which the distance is measured from two images havingextended depths of field, using the imaging apparatus 10 according tothe embodiment of the present invention. Note that the description belowis given assuming that a method of changing a focal position duringexposure is used as a technique to extend the depth of field.

First, the imaging unit 11 captures the first image 31 a and the secondimage 31 b which have mutually different focal ranges. Specifically,during the exposure of the first image 31 a and the second image 31 b,the focal range control unit 12 causes the image plane position to moveat a constant speed as shown in FIG. 4. At this time, the focal rangecontrol unit 12 causes the imaging unit 11 to capture the first image 31a, causing the image plane position to move from v1 to v3 (Step S101).Next, the focal range control unit 12 causes the focal position to moveby a predetermined distance (Step S102). Specifically, the focal rangecontrol unit 12 causes a capturing position to move from v3 to v4 at theconstant speed during non-exposure period. After that, the focal rangecontrol unit 12 causes the imaging unit 11 to capture the second image31 b, causing the image plane position to move from v4 to v2 at theconstant speed (Step S103). As shown in FIG. 5, the first image 31 athus captured has the uniform blur within the focal range of the firstimage 31 a that corresponds to image plane positions v1 to v3, and hasthe blur degree according to the distance from the focal range of thefirst image 31 a in a position out of the image plane positions v1 tov3. Similarly, the second image 31 b has the uniform blur within thefocal range of the second image 31 b that corresponds to the image planepositions v4 to v2, and has the blur degree according to the distancefrom the focal range of the second image 31 b in a position out of theimage plane positions v4 to v2. A blur that is uniform with respect tothe subject distance contributes to generation of the reference image32, and a blur according to the subject distance contributes to thedistance measurement by DFD, respectively.

Note that v3 and v4 can be set at an arbitrary position between v1 andv2, but it is preferable to set the distance between v1 and v3 and thedistance between v4 and v2 are equal to each other in order to allow thefirst image 31 a and the second image 31 b to have the sameexposure-time period. The first image 31 a and the second image 31 bhave the same exposure-time period, allowing the first and second images31 a and 31 b to include noise in the same level. This improves accuracyof distance calculation.

In addition, as shown in FIG. 4, the focal range control unit 12 causesthe focal position to move at a constant speed during a time periodwhich covers the exposure of the first image 31 a, the non-exposure, andthe exposure of the second image 31 b. In other words, the focal rangecontrol unit 12 causes the image plane position to change at a constantspeed during a time period from initiation to completion of thecapturing of the first image 31 a and the second image 31 b by theimaging unit 11. This makes it possible to easily control the change ofthe focal position.

Note that first image 31 a and the second image 31 b may be different ina movement speed of the focal position during exposure.

Next, the reference image generation unit 13 generates the referenceimage 32 from the first image 31 a and the second image 31 b (StepS104). From Expression 8, the reference image 32 can be calculated bydeconvolution of PSF into the captured image. Although the subjectdistance d(x, y) should essentially be known in order to calculate thereference image 32 accurately, PSF is constant with respect to thesubject distance within the extended depth of field because the depthsof field of the first image 31 a and the second image 31 b are extended.

Here, the reference image generation unit 13 calculates an average ofthe first image 31 a and the second image 31 b to generate an averageimage. As is clear from FIG. 4, the average image is an image having anapproximately uniform depth of field across the entire range from v1 tov2 except for the range from v3 to v4. In other words, the average imageis approximately equivalent to an image having the uniform blur acrossthe entire range from v1 to v2. Accordingly, the reference imagegeneration unit 13 performs a deconvolution operation into the averageimage by a single type of PSF corresponding to the uniform blur, therebygenerating the reference image 32 focused in the entire range from v1 tov2.

If the first image 31 a and the second image 31 b are different inlength of the exposure-time period or movement speed of the focalpositions, the images can be treated, in the similar manner as above, bycalculating a weighted average such that a constant weight is given tothe subject distance.

In addition, a known technique such as Wiener Filter can be used fordeconvolution algorithm.

Lastly, the distance measurement unit 14 calculates a distance map d(x,y) in accordance with Expression 9, from the captured first image 31 a,second image 31 b, and the reference image 32 which is generated in stepS104 (Step S105). Specifically, the distance measurement unit 14calculates, in accordance with Expression 9, a distance map using thefirst image 31 a and a distance map using the second image 31 b. Next,the distance measurement unit 14 calculates one distance map by mergingthese maps (Step S106). For the merger, it is possible to use atechnique such as calculating a simple average of the distance mapscalculated from the respective images.

Note that the distance measurement may be performed by Fouriertransforming Expressions 8 and 9, and comparing the reference image witheach of the captured images in the frequency domain. In the frequencydomain, the convolution operations in Expressions 8 and 9 aretransformed into multiplication, so that distance measurement can beperformed at a higher speed.

As described above, the imaging unit 10 according to the embodiment cangenerate the reference image 32 using only the first image 31 a and thesecond image 31 b. Accordingly, the imaging device 10 can perform thedistance measurement accurately with a small number of images. Since thefocus change during the exposure can be realized by diverting a standardautofocus mechanism, no special mechanism is required.

Note that an example of using two images has been described above, butthree or more images may be captured. In this case, an out-of-focalrange is provided between each of the focal ranges of the images, andthe length of each focal range is set to equal to each other, so thatthe images can be treated in the same manner as in the case of using twoimages. In this context, the reference image is calculated from anaverage image of all the captured images.

Although an example using the average image of the plurality of imagesis described in the above, the same processing as the example can becarried out using an addition image prepared by adding up a plurality ofimages.

In addition, in the above description, an image generated by performingthe deconvolution operation on the average image by PSF is used as thereference image 32, but an image which has the approximately uniformblur in the entire range from v1 to v2 can be used as the referenceimage 32. For example, the aforementioned average image or the additionimage may be used as the reference image 32.

[Simulation Estimation Result]

Hereinafter, an advantage of having a discontinuous focal range isdescribed. It is assumed that a value of an appropriate distance at aposition (x, y) is D, and a value of an inappropriate distance is D′.According to Expression 9, the greater the difference between theresults of convolution of h(x, y, d) and h(x, y, D′) into the referenceimage R, the easier it becomes to determine whether the distance isappropriate or not. This will be verified by simulation below.

A scene is considered which has a texture as shown in FIG. 6A and inwhich the subject distance has a stepped pattern made up of 20 steps interms of the distance up to the subject as shown in FIG. 6B. It isassumed that a darker portion in the figure indicates a farther subjectdistance.

A lens assumed in the simulation has performance of 9 mm focal distanceand 1.4 F-number.

Next, a notation of the focal range is described. It is understoodwhether or not each of the 20 steps in the subject distance is in focusis represented by a 20-digit number. For example, [1, 0, 1, 1, 1, 0, 0,0, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] represents that steps 1, 3, 4, 5,and 10 are in focus, starting from the farthest one.

First, FIG. 7A shows a result of the distance measurement by DFD fromtwo images having continuous focal ranges (without a space).Specifically, the two images includes an image having the focal range of[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] (first totenth steps) and an image having the focal range of [0, 0, 0, 0, 0, 0,0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1] (eleventh to twentieth steps).At this time, the subject distance can be calculated according toExpression 10 and Expression 11 below.

$\begin{matrix}{\mspace{79mu}\lbrack {{Math}.\mspace{14mu} 8} \rbrack} & \; \\{{{d( {x,y} )} = {\underset{d}{\arg\;\min}( {{{f^{- 1}( {{\hat{F}{H_{1}(d)}} - F_{1}} )}} + {{f^{- 1}( {{\hat{F}{H_{2}(d)}} - F_{2}} )}}} )}}\mspace{79mu}( {{d = d_{1}},d_{2},\ldots\mspace{14mu},d_{n}} )} & {{Expression}\mspace{14mu} 10} \\{\mspace{79mu}{\hat{F} = \frac{{F_{1}{H_{1}^{*}(d)}} + {F_{2}{H_{2}^{*}(d)}}}{{{H_{1}(d)}{H_{1}^{*}(d)}} + {{H_{2}(d)}{H_{2}^{*}(d)}} + ɛ}}} & {{Expression}\mspace{14mu} 11}\end{matrix}$

However, F1 and F2 represent the frequency components of the first andsecond images, respectively; H1 and H2 represent frequency components ofPSF corresponding to the focal ranges of the first and the secondimages, respectively; and H1* and H2* are complex conjugates of H1 andH2, respectively; ε represents a minor value to prevent division byzero; and f⁻¹ represents inverse Fourier transform.

Meanwhile, FIG. 7B shows a result of the distance measurement from twoimages having a discontinuous focal range, in the same manner with thedistance measurement from the two images having continuous focal ranges.Specifically, these two images are an image having the focal range of[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] (first totenth steps) and an image having the focal range of [0, 0, 0, 0, 0, 0,0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1,] (twelfth to twentieth steps).This shows that the eleventh step is the non-focal range.

Here, the closer to the texture shown in FIG. 6B the distancemeasurement is, the higher the accuracy is. FIG. 7A shows that theresult of the measurement is different from the texture shown in FIG. 6Bparticularly on the right side in which a little texture appears,whereas FIG. 7B shows an improved measurement result in comparison withthat shown in FIG. 7A.

FIGS. 8A and 8B are respectively a table and a graph, each of whichshows a difference between the measurement result and appropriate datawith respect to the space between the focal ranges. Root mean square(RMS) is used for the difference between the measurement result and theappropriate data. The closer to 0 a value of the RMS is, the higher themeasurement accuracy is. FIG. 8A is a table which represents the RMS ofthe distance measurement result with respect to the space between focalranges. FIG. 8B is a diagram which represents the RMS in a bar graph.

As shown in FIGS. 8A and 8B, a smallest combination of RMS is obtainedwhen the out-of-focus range is set in three steps from tenth to twelfthsteps. The measurement accuracy is higher in the discontinuous focalranges than that in the continuous focal ranges. However, an extremelybroad space causes the measurement accuracy to decrease. Specifically,the space is preferably provided across four steps or fewer, morepreferably, in two to four steps.

Here, one step corresponds to about 50 μm in the image plane position.In other words, it is preferable that the image plane positioncorresponding to the space is 200 μm (corresponding to four steps) orless, more preferably, from 100 μm (corresponding to two steps) to 200μm (corresponding to four steps).

However, since characteristics of the blur vary depending on a lens,this space is not necessarily optimal in all cases.

In the above description, the images used for DFD is capturedindividually. If two images are simultaneously captured, time requiredfor the capturing can be reduced to half of that for the individualimage-capturing. Hereinafter, variations of the present embodiment aredescribed.

<Variation 1>

FIG. 9A is a diagram which shows a configuration of the imaging unit 11according to Variation 1, while FIG. 9B is a graph which shows change infocal positions according to the Variation 1.

As shown in FIG. 9A, the imaging unit 11 includes a lens 21, two imagingdevices 22 a and 22 b, a beam splitter 23 for splitting light, and amirror 24 for bending an optical path the light. The two imaging devices22 a and 22 b are arranged to have different optical path lengths fromthe lens 21. In the configuration shown in FIG. 9A, the imaging device22 b has a longer optical path than the imaging device 22 a by Δv. Thebeam splitter 23 splits the light from the lens 21 toward the twoimaging devices 22 a and 22 b.

The focal range control unit 12 causes the imaging device 22 a and theimaging device 22 b to simultaneously capture the first image 31 a andthe second image 31 b respectively, changing the focal positions of thetwo imaging devices 22 a and 22 b simultaneously during the same timeperiod. In such an optical system, specifically, when images aresimultaneously captured using two imaging devices 22 a and 22 b with thefocal position being changed at a constant speed during exposure, imagescan be obtained which have focal positions constantly displaced fromeach other by Δv in the imaging devices 22 a and 22 b, as shown in FIG.9B. In this context, Δv is set to be equal to a distance from v1 to v4,so that two images are simultaneously captured which are not in focus ina distance from v3 to v4.

Note that as a unit to bend the optical path, a prism may be usedinstead of a mirror. Alternatively, as shown in FIG. 10 described later,a configuration without a mirror or prism may be employed.

<Variation 2>

FIG. 10 is a diagram which shows a configuration of the imaging unit 11according to Variation 2. FIGS. 11A and 11B are diagrams each of whichshows operation of the imaging unit 11. FIG. 12 is a graph which shows achange in focal positions according to Variation 2.

The imaging unit 11 shown in FIG. 10 includes the lens 21, the twoimaging devices 22 a and 22 b, an aperture 25, and a movable mirror 26.The movable mirror 26 is arranged in the optical path from the lens 21to the two imaging devices 22 a and 22 b, and serves as an optical axischanging unit for changing an optical-axis direction of a light beamfrom the lens 21. In addition, the movable mirror 26 functions as aselection unit for causing the light beam from the lens 21 toselectively enter one of the two imaging devices 22 a and 22 b.

Furthermore, two imaging devices 22 a and 22 b have an equaloptical-path length from the lens 21.

The movable mirror 26 is a galvanometer mirror or a MEMS mirror, forexample. The movable mirror 26 has a function to guide the light beam toone of the two imaging devices 22 a and 22 b, as shown in FIGS. 11A and11C.

The aperture 25 blocks the light beam during operation of the movablemirror 26 to switch the imaging devices one of which the light beam isto reach, as shown in FIG. 11B.

The focal range control unit 12 selects two imaging devices 22 a and 22b sequentially, and causes the movable mirror 26 to allow the light beamto selectively enter the selected imaging device, thereby causing eachof the two imaging devices to capture a corresponding one of the twoimages. Specifically, in such an optical system, the focal position ischanged at a constant speed during exposure while at the same time thelight beam is directed to one of the imaging devices 22 a and 22 b bythe movable mirror 26. The light beam reaches the imaging device 22 awhen the focal position is in the range from v1 to v3, while the lightbeam reaches the imaging device 22 b when the focal position is in therange from v4 to v2. Accordingly, as shown in FIG. 12, the first image31 a and the second image 31 b have discontinuous focal ranges.

Here, in a configuration where the first image 31 a and the second image31 b are captured using a single imaging device, a time period forreading data and the like is required before initiation of capturing ofthe second image 31 b after completion of the capturing of the firstimage 31 a. In contrast, in a configuration of Variation 2, such timeperiod is not required, so that a non-exposure time period can beshortened.

Note that the optical path in Variation 2 is not limited only to theconfiguration shown in FIG. 10, but may employ an arbitrary optical pathas long as the plural imaging devices have the same optical length.

<Other Variations>

Although the present invention is described in accordance with theaforementioned embodiments, it is needless to say that the presentinvention is not limited to the aforementioned embodiments. The presentinvention also involves the following.

(1) A part or all of components constituting the aforementionedrespective devices may be formed, specifically, as a computer systemincluding a microprocessor, a ROM, a RAM, a hard disc unit, a displayunit, a keyboard, a mouse, and so on. The RAM or the hard disc unitstores a computer program. The microprocessor operates in accordancewith the computer program, so that each of the devices accomplishes itsfunction. Here, the computer program is, for accomplishing apredetermined function, configured by combining a plurality ofinstruction codes indicating instructions for a computer.

(2) A part or all of the components constituting each of the abovedevices may be formed by a single System-LSI (Large-Sale Integration)circuit. The System LSI is a super multifunction LSI manufactured byintegrating a plurality of constituent units on a single chip, and is,specifically, a computer system including a microprocessor, a ROM, aRAM, and so on. The RAM stores a computer program. The microprocessoroperates in accordance with the computer program, so that the System-LSIaccomplishes its function.

(3) A part or all of the components constituting each of the devices maybe formed as an IC card which is detachable from each of the devices ora single module. The IC card or the module is a computer systemincluding a microprocessor, a ROM, a RAM, and so on. The IC card or themodule may include the super multifunction LSI. The microprocessoroperates in accordance with the computer program, so that the IC card orthe module accomplishes its function. The IC card or the module may havetamper resistance.

(4) The present invention may be in the form of the method describedabove. In addition, the present invention may be a computer programwhich realizes the method by a computer, or may be digital signalsincluding the computer program.

The present invention may also be realized by storing the computerprogram or the digital signal in a computer readable recording medium,such as a flexible disc, a hard disc, a CD-ROM, an MO, a DVD, a DVD-ROM,a DVD-RAM, a Blue-ray disc (BD), a semiconductor memory, and so on.Alternatively, the present invention may also include the digital signalrecorded in these recording media.

The present invention may also be realized by transmission of theaforementioned computer program or digital signal via an electrictelecommunication line, a wireless or wired communication line, anetwork represented by the Internet, a data broadcast, and so on.

The present invention may also be a computer system including amicroprocessor and a memory, in which the memory stores theaforementioned computer program, and the microprocessor operates inaccordance with the computer program.

Furthermore, the program or the digital signal may be stored in therecording medium so as to be transferred, or the program or the digitalsignal may be transferred via the network or the like so as to beexecuted by another independent computer system.

(5) The above embodiments and modifications may be combined arbitrarily.

All numerals used in the above represent examples for specificallydescribing the present invention, and the present invention is notlimited to the numerals.

Furthermore, segmentation of functional blocks in the block diagram isan example, so that a plurality of functional blocks may be realized asa single functional block, a single functional block may be segmentedinto a plurality of blocks, or a part of function may be transferred toanother functional block. Furthermore, the function of a plurality offunctional blocks having functions similar to one another may beprocessed by a single hardware or software in parallel or in atime-division system.

A flow of execution of the above steps is an example for specificallydescribing the present invention, and thus, a flow other than the abovemay be employed. A part of the steps may be executed simultaneously (inparallel) with other steps.

All possible variations which include changes added by a person skilledin the art in his/her conceivable range may be involved in the presentinvention as long as the variations are not depart from the principlesof the present invention.

INDUSTRIAL APPLICABILITY

The present invention is applicable to imaging apparatuses which includea lens system, particularly to a monocular imaging apparatus.

REFERENCE SIGNS LIST

-   10 Imaging apparatus-   11 Imaging unit-   12 Focal range control unit-   13 Reference image generation unit-   14 Distance measurement unit-   21 lens-   22, 22 a, 22 b Imaging device-   23 Beam splitter-   24 Mirror-   25 Aperture-   26 Movable mirror-   31 a First image-   31 b Second image-   32 Reference image

The invention claimed is:
 1. An imaging apparatus comprising: an imagingunit configured to image a subject to generate an image; a focal rangecontrol unit configured to cause the imaging unit to capture n imageshaving mutually different focal ranges, changing a focal position of theimaging unit, where n is an integer greater than or equal to 2; areference image generation unit configured to generate, using the nimaged, a reference image to be used as a blur standard; and a distancemeasurement unit configured to measure a distance to the subject basedon a difference in blur degrees between the reference image and each ofthe n images, wherein the focal ranges of the n images are independentof each other, and an out-of-focus space is provided between the focalranges.
 2. The imaging apparatus according to claim 1, wherein theimaging unit includes: an imaging device; and a lens which collectslight into the imaging device, and the focal range control unit isconfigured to change the focal position by changing a distance betweenthe imaging device and the lens at a constant speed.
 3. The imagingapparatus according to claim 2, wherein the n images have a sameexposure-time period.
 4. The imaging apparatus according to claim 2,wherein the focal range control unit is configured to change thedistance between the imaging device and the lens at the constant speedduring a time period from initiation to completion of the imaging of then images by the imaging unit.
 5. The imaging apparatus according toclaim 1, wherein the imaging unit includes: a lens; n imaging devicesarranged to have mutually different optical path lengths from the lens;and a beam splitter which splits a light beam from the lens into lightbeams for the respective n imaging devices, and the focal range controlunit is configured to cause the n imaging devices to simultaneouslycapture the respective n images, changing the focal positions of the nimaging devices simultaneously, during a same time period.
 6. Theimaging apparatus according to claim 1, wherein the imaging unitincludes: a lens; n imaging devices; and a selection unit configured toallow a light beam from the lens to selectively enter one of the nimaging devices, and the focal range control unit is configured tosequentially select the n imaging devices, and to cause the selectionunit to allow the light beam to selectively enter the selected imagingdevice, thereby causing each of the n imaging devices to capture acorresponding one of the n images.
 7. The imaging apparatus according toclaim 1, wherein the reference image generation unit is configured togenerate an average image of the n images, and to generate the referenceimage using the average image.
 8. The imaging apparatus according toclaim 7, wherein the reference image generation unit is configured togenerate the reference image by performing, on the average image, adeconvolution operation by a single point spread function.
 9. A distancemeasurement method performed using an imaging apparatus which includesan imaging unit configured to image a subject to generate an image, thedistance measurement method comprising: causing the imaging unit tocapture n images having mutually different focal ranges, changing afocal position of the imaging unit, where n is an integer greater thanor equal to 2; generating, using the n images, a reference image to beused as a blur standard; and measuring a distance to the subject basedon a difference in blur degrees between the reference image and each ofthe n images, wherein the focal ranges of the n images are independentof each other, and an out-of-focus space is provided between the focalranges.
 10. A non-transitory computer-readable recording medium whichholds a program for causing a computer to execute the distancemeasurement method according to claim 9.